A Closed-Loop Perspective on Symbiotic Human-Computer Interaction

This paper is concerned with how people interact with an emergent form of technology that is capable of both monitoring and affecting the psychology and behaviour of the user. The current relationship between people and computer is characterised as asymmetrical and static. The closed-loop dynamic of physiological computing systems is used as an example of a symmetrical and symbiotic HCI, where the central nervous system of the user and an adaptive software controller are engaged in constant dialogue. This emergent technology offers several benefits such as: intelligent adaptation, a capacity to learn and an ability to personalise software to the individual. This paper argues that such benefits can only be obtained at the cost of a strategic reconfiguration of the relationship between people and technology - specifically users must cede a degree of control over their interaction with technology in order to create an interaction that is active, dynamic and capable of responding in a stochastic fashion. The capacity of the system to successfully translate human goals and values into adaptive responses that are appropriate and effective at the interface represents a particular challenge. It is concluded that technology can develop lifelike qualities (e.g. complexity, sentience, freedom) through sustained and symbiotic interaction with human beings. However, there are a number of risks associated with this strategy as interaction with this category of technology can subvert skills, self-knowledge and the autonomy of human user

Signal Processing of Multimodal Mobile Lifelogging Data towards Detecting Stress in Real-World Driving

Stress is a negative emotion that is part of everyday life. However, frequent episodes or prolonged periods of stress can be detrimental to long-term health. Nevertheless, developing self-awareness is an important aspect of fostering effective ways to self-regulate these experiences. Mobile lifelogging systems provide an ideal platform to support self-regulation of stress by raising awareness of negative emotional states via continuous recording of psychophysiological and behavioural data. However, obtaining meaningful information from large volumes of raw data represents a significant challenge because these data must be accurately quantified and processed before stress can be detected. This work describes a set of algorithms designed to process multiple streams of lifelogging data for stress detection in the context of real world driving. Two data collection exercises have been performed where multimodal data, including raw cardiovascular activity and driving information, were collected from twenty-one people during daily commuter journeys. Our approach enabled us to 1) pre-process raw physiological data to calculate valid measures of heart rate variability, a significant marker of stress, 2) identify/correct artefacts in the raw physiological data and 3) provide a comparison between several classifiers for detecting stress. Results were positive and ensemble classification models provided a maximum accuracy of 86.9% for binary detection of stress in the real-world

Detecting Negative Emotions During Real-Life Driving via Dynamically Labelled Physiological Data

Driving is an activity that can induce significant levels of negative emotion, such as stress and anger. These negative emotions occur naturally in everyday life, but frequent episodes can be detrimental to cardiovascular health in the long term. The development of monitoring systems to detect negative emotions often rely on labels derived from subjective self-report. However, this approach is burdensome, intrusive, low fidelity (i.e. scales are administered infrequently) and places huge reliance on the veracity of subjective self-report. This paper explores an alternative approach that provides greater fidelity by using psychophysiological data (e.g. heart rate) to dynamically label data derived from the driving task (e.g. speed, road type). A number of different techniques for generating labels for machine learning were compared: 1) deriving labels from subjective self-report and 2) labelling data via psychophysiological activity (e.g. heart rate (HR), pulse transit time (PTT), etc.) to create dynamic labels of high vs. low anxiety for each participant. The classification accuracy associated with both labelling techniques was evaluated using Linear Discriminant Analysis (LDA) and Support Vector Machines (SVM). Results indicated that classification of driving data using subjective labelled data (1) achieved a maximum AUC of 73%, whilst the labels derived from psychophysiological data (2) achieved equivalent performance of 74%. Whilst classification performance was similar, labelling driving data via psychophysiology offers a number of advantages over self-reports, e.g. implicit, dynamic, objective, high fidelity

Effects of self-directed and other-directed introspection and emotional valence on activation of the rostral prefrontal cortex during aesthetic experience

The medial area of the rostral prefrontal cortex (rPFC) has been implicated in self-relevant processing, autobiographical memory and emotional processing, including the processing of pleasure during aesthetic experiences. The goal of this study was to investigate changes in rPFC activity using functional near-infrared spectroscopy (fNIRS) in response to affective stimuli viewed in a self-relevant or other-relevant context. Positive and negative images were displayed to 20 participants under two viewing conditions where participants were asked to think of their own emotions (self) or think about the emotions of the artist who created the work (other). The results revealed an increase of HbO when participants viewed images during the other-condition compared to the self-condition. It was concluded that viewing stimuli from the perspective of another was associated with an increase of cognitive demand. The analysis of deoxygenated haemoglobin (HHb) at right hemispheric areas revealed that activation of the rPFC during the other-condition was specific to the negative images. When images were viewed from the perspective of the self, activation of the rPFC significantly increased at the right-medial area of the rPFC for positive images. Our findings indicate that the influence of valence on rPFC activation during aesthetic experience is contingent on the context of the viewing experience and there is a bias towards positive emotion when images are viewed from the context of the self

A Mobile Lifelogging Platform to Measure Anxiety and Anger During Real-Life Driving

The experience of negative emotions in everyday life, such as anger and anxiety, can have adverse effects on long-term cardiovascular health. However, objective measurements provided by mobile technology can promote insight into this psychobiological process and promote self-awareness and adaptive coping. It is postulated that the creation of a mobile lifelogging platform can support this approach by continuously recording personal data via mobile/wearable devices and processing this information to measure physiological correlates of negative emotions. This paper describes the development of a mobile lifelogging system that measures anxiety and anger during real-life driving. A number of data streams have been incorporated in the platform, including cardiovascular data, speed of the vehicle and first-person photographs of the environment. In addition, thirteen participants completed five days of data collection during daily commuter journeys to test the system. The design of the system hardware and associated data streams are described in the current paper, along with the results of preliminary data analysis

Assessment of threat and negativity bias in virtual reality

Negativity bias, i.e., tendency to respond strongly to negative stimuli, can be captured via behavioural and psychophysiological responses to potential threat. A virtual environment (VE) was created at room-scale wherein participants traversed a grid of ice blocks placed 200 m above the ground. Threat was manipulated by increasing the probability of encountering ice blocks that disintegrated and led to a virtual fall. Participants interacted with the ice blocks via sensors placed on their feet. Thirty-four people were recruited for the study, who were divided into High (HN) and Low (LN) Neuroticism groups. Movement data were recorded alongside skin conductance level and facial electromyography from the corrugator supercilii and zygomaticus major. Risk-averse behaviours, such as standing on ‘safe’ blocks and testing blocks prior to movement, increased when threat was highest. HN individuals exhibited more risk-averse behaviour than the LN group, especially in the presence of high threat. In addition, activation of the corrugator muscle was higher for HN individuals in the period following a movement to an ice block. These findings are discussed with respect to the use of room-scale VE as a protocol for emotion induction and measuring trait differences in negativity bias within VR

A Neuroergonomics Approach to Mental Workload, Engagement and Human Performance

The assessment and prediction of cognitive performance is a key issue for any discipline concerned with human operators in the context of safety-critical behavior. Most of the research has focused on the measurement of mental workload but this construct remains difficult to operationalize despite decades of research on the topic. Recent advances in Neuroergonomics have expanded our understanding of neurocognitive processes across different operational domains. We provide a framework to disentangle those neural mechanisms that underpin the relationship between task demand, arousal, mental workload and human performance. This approach advocates targeting those specific mental states that precede a reduction of performance efficacy. A number of undesirable neurocognitive states (mind wandering, effort withdrawal, perseveration, inattentional phenomena) are identified and mapped within a two-dimensional conceptual space encompassing task engagement and arousal. We argue that monitoring the prefrontal cortex and its deactivation can index a generic shift from a nominal operational state to an impaired one where performance is likely to degrade. Neurophysiological, physiological and behavioral markers that specifically account for these states are identified. We then propose a typology of neuroadaptive countermeasures to mitigate these undesirable mental states

The Influence of Game Demand on Distraction from Experimental Pain: A fNIRS Study

Video games are the most effective form of distraction from procedural pain compared to other distraction techniques, such as watching television or reading a book (Hussein, 2015). The degree of cognitive engagement with the game is a strong influence on the capacity of game-playing to distract from pain. By increasing game demand to a level that demands maximum levels of attention, it is possible to optimise distraction from pain; however, if the game becomes too difficult, it will fail to act as a distraction

A Lifelogging Platform Towards Detecting Negative Emotions in Everyday Life using Wearable Devices

Repeated experiences of negative emotions, such as stress, anger or anxiety, can have long-term consequences for health. These episodes of negative emotion can be associated with inflammatory changes in the body, which are clinically relevant for the development of disease in the long-term. However, the development of effective coping strategies can mediate this causal chain. The proliferation of ubiquitous and unobtrusive sensor technology supports an increased awareness of those physiological states associated with negative emotion and supports the development of effective coping strategies. Smartphone and wearable devices utilise multiple on-board sensors that are capable of capturing daily behaviours in a permanent and comprehensive manner, which can be used as the basis for self-reflection and insight. However, there are a number of inherent challenges in this application, including unobtrusive monitoring, data processing, and analysis. This paper posits a mobile lifelogging platform that utilises wearable technology to monitor and classify levels of stress. A pilot study has been undertaken with six participants, who completed up to ten days of data collection. During this time, they wore a wearable device on the wrist during waking hours to collect instances of heart rate (HR) and Galvanic Skin Resistance (GSR). Preliminary data analysis was undertaken using three supervised machine learning algorithms: Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA) and Decision Tree (DT). An accuracy of 70% was achieved using the Decision Tree algorithm